Skip to content

[refactor](storage) Unify BE and Recycler object clients - #66350

Open
sollhui wants to merge 9 commits into
apache:masterfrom
sollhui:agent/unify-be-recycler-obj-client
Open

[refactor](storage) Unify BE and Recycler object clients#66350
sollhui wants to merge 9 commits into
apache:masterfrom
sollhui:agent/unify-be-recycler-obj-client

Conversation

@sollhui

@sollhui sollhui commented Aug 1, 2026

Copy link
Copy Markdown
Contributor

1. What does this PR do?

BE and Cloud Recycler previously maintained separate object-storage abstractions and separate S3/Azure implementations. Although both sides called the same cloud-provider SDKs, credential construction, error conversion, metrics, pagination, batch deletion, and compatibility behavior were duplicated and could evolve differently.

This PR consolidates the implementation under common/cpp/client and exposes one ObjStorageClient facade to upper layers:

  • ObjStorageClient owns backend-independent orchestration and is the only complete client used by BE and Recycler call sites.
  • ObjStorageRateLimitPolicy keeps BE- and Recycler-specific admission behavior injectable without coupling common code to either environment.
  • ObjStorageBackend is the storage implementation boundary, implemented by S3ObjStorageBackend and AzureObjStorageBackend.
  • Shared request/response types, page-based listing, upper-layer lazy iteration, recursive deletion, backend batch capabilities, credentials, metrics, and error conversion live in the common layer.
  • Request admission remains attached to actual backend work: one GET admission per list page and one PUT admission per backend-sized delete batch.

Before the refactor, BE and Recycler reached the cloud SDKs through parallel stacks:

                                     BEFORE

  +---------------------- BE ----------------------+     +------------------- Recycler -------------------+
  |  BE call sites -> ObjClientHolder              |     |  Recycler call sites -> S3Accessor            |
  |                       |                        |     |                       |                        |
  |                       v                        |     |                       v                        |
  |  BE limiter + separate S3 / Azure clients      |     |  Recycler limiter + separate S3 / Azure clients|
  |  credentials / listing / recursive deletion    |     |  credentials / listing / recursive deletion    |
  |  error conversion / metrics                    |     |  error conversion / metrics                    |
  +------------------------+-----------------------+     +------------------------+-----------------------+
                           |                                                        |
                           v                                                        v
                    AWS SDK / Azure SDK                                      AWS SDK / Azure SDK

After the refactor, BE and Recycler stay on the left and right while the shared facade and backend components are centered below them:

                                      AFTER

  +---------------------- BE ----------------------+     +------------------- Recycler -------------------+
  |  BE call sites -> ObjClientHolder / factory    |     |  Recycler call sites -> S3Accessor / adapter  |
  +-----------------------------+------------------+     +------------------+-----------------------------+
                                |                                           |
                                +-------------------+-----------------------+
                                                    |
                                                    v
                          +--------------------------------------------------+
                          |            ObjStorageClient facade               |
                          |                                                  |
                          |  +----------------------+  +-------------------+  |
                          |  | RateLimitPolicy      |  | ObjStorageBackend |  |
                          |  | - BE policy          |  |        |          |  |
                          |  | - Recycler policy    |  |   +----+----+     |  |
                          |  +----------------------+  |   |         |     |  |
                          |                            |   v         v     |  |
                          |                            | S3 Backend Azure   |  |
                          |                            | Backend            |  |
                          |                            +---+---------+------+  |
                          +--------------------------------|---------|---------+
                                                           v         v
                                                        AWS SDK   Azure SDK

This design keeps current BE and Recycler production call sites behind the policy-bearing facade, avoiding accidental policy bypass. Backend code implements cloud mechanics; common orchestration and policy dispatch remain in the facade.

2. How are the different behaviors unified?

Behavior BE before Recycler before Unified behavior
Client API BE-specific doris::io::ObjStorageClient and eager list results Recycler-specific client and iterator APIs One doris::ObjStorageClient facade and one set of request/response types. doris::io aliases keep BE call sites source-compatible; Recycler adapters preserve its integer-facing API.
Backend implementation Separate BE S3/Azure clients Separate Recycler S3/Azure clients S3ObjStorageBackend and AzureObjStorageBackend are shared by both callers.
Rate limiting BE owned QPS/bytes limiters and bucket-selection rules Recycler owned its limiter and fault injection Each environment injects an ObjStorageRateLimitPolicy; the facade performs admission immediately before backend work. Each list page and each backend-sized delete batch is admitted independently.
Error model BE status codes plus HTTP metadata on selected paths Recycler-specific return codes and messages ObjectStorageResponse carries a Doris status code and preserves an HTTP code and request ID when available, while adapters preserve caller-facing behavior such as the Recycler 0/1/negative exists contract.
Listing BE eagerly collected all pages Recycler exposed a lazy iterator ObjStorageClient::list_objects returns one fixed-size ObjectStorageListPage. For an admitted page request, one Client call maps to one Backend call and one SDK request. The upper ObjectListIterator owns the continuation token and requests the next page only after its cached page is consumed.
End of listing Eager completion was represented by a finished vector Iterator completion used empty/false results END_OF_FILE is an internal upper-iterator sentinel and next() converts it to a successful empty result. Backend NOT_FOUND remains a real error.
Missing S3 prefix S3-compatible NoSuchKey handling existed in the BE path The compatibility behavior was maintained separately The shared S3 backend preserves NoSuchKey-as-empty behavior once for both callers.
Direct batch deletion Backend limits were embedded in separate implementations Recycler maintained its own batching The facade splits by backend capability (1000 for S3, 256 for Azure), acquires one PUT admission per backend-sized batch, and keeps defensive bounds in each backend.
Recursive deletion Separate implementations and grouping behavior Recycler supported expiration filtering and parallel execution The facade owns one shared listing/filtering/grouping/error-propagation flow. Every list page goes through the policy-bearing one-page API, and every delete task acquires PUT admission immediately before its backend request. Recycler injects an executor adapter backed by SyncExecutor; BE uses the synchronous fallback.
AWS credentials BE built static/default/role providers in its factory Recycler maintained another construction path AwsCredentialFactory implements static credentials, default provider chains, role ARN, and external ID once while callers retain their prior empty-credential behavior.
Azure credentials BE and Recycler built shared-key clients separately Separate construction and credential retention AzureAuthFactory creates the container client and shared-key credential for both. The BE factory passes TLS diagnostic context into the shared backend, whose common conversion helpers append it to matching TLS/CA failures.
Metrics and latency Duplicated stopwatch and failure-recording paths Separate helpers recorded equivalent data Backends use the shared client_bvar::ScopedLatency timer and common failure metrics. The previous BE 5-second slow-request logging behavior is retained for S3 uploads.
Backend-specific APIs Shared interfaces forced unrelated test stubs Lifecycle, versioning, and multipart-abort were Recycler-oriented The backend boundary supplies default not-supported responses and implements supported APIs without leaking backend details to callers.

3. Design boundaries and follow-ups

  • Upper layers keep std::shared_ptr<ObjStorageClient>; they do not store ObjStorageBackend directly.
  • BE and Recycler own their policy configuration, but both use the same facade dispatch and the same S3/Azure backends.
  • The upper ObjectListIterator performs lazy iteration by repeatedly calling the one-page Client API. Each page request attempts one facade admission and, once admitted, issues one SDK request; reading objects already cached in that page performs no network request.
  • A public delete_objects call may split input according to backend capability, but each resulting backend batch acquires its own PUT admission before issuing one SDK request.
  • Recursive deletion composes the same one-page list and backend-sized delete operations. Consequently every actual list page and delete batch is counted by the injected rate-limit policy; Recycler may execute delete tasks in parallel through SyncExecutor.

4. How was this PR tested?

  • Local compilation and unit tests were not run, as requested during development.
  • Changed C++ files were formatted.
  • git diff --check passed.
  • CI/buildall has been triggered; refer to the PR checks for the latest status.

@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@sollhui

sollhui commented Aug 1, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

@sollhui

sollhui commented Aug 1, 2026

Copy link
Copy Markdown
Contributor Author

/review

@sollhui
sollhui force-pushed the agent/unify-be-recycler-obj-client branch from f30abce to dc25d3e Compare August 1, 2026 09:44
@github-actions

github-actions Bot commented Aug 1, 2026

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: Codex completed, but no new pull request review was submitted for the current head SHA.
Workflow run: https://github.com/apache/doris/actions/runs/30693205073

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

Comment thread common/cpp/client/obj_storage_client.cpp
@sollhui

sollhui commented Aug 3, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

@sollhui

sollhui commented Aug 3, 2026

Copy link
Copy Markdown
Contributor Author

/review

@sollhui

sollhui commented Aug 3, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

@sollhui

sollhui commented Aug 3, 2026

Copy link
Copy Markdown
Contributor Author

/review

@github-actions

github-actions Bot commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: Codex completed, but no new pull request review was submitted for the current head SHA.
Workflow run: https://github.com/apache/doris/actions/runs/30794530392

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

@github-actions

github-actions Bot commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: Review step was failure (possibly timeout or cancelled)
Workflow run: https://github.com/apache/doris/actions/runs/30797931848

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 0.00% (0/1) 🎉
Increment coverage report
Complete coverage report

@sollhui
sollhui force-pushed the agent/unify-be-recycler-obj-client branch from 1fab2ee to 4ea09ca Compare August 3, 2026 13:48
@sollhui

sollhui commented Aug 3, 2026

Copy link
Copy Markdown
Contributor Author

/review

@sollhui

sollhui commented Aug 3, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Static-only review of the full authoritative diff found six issues that should be addressed before merge (five P1, one P2).

Review-cycle status: incomplete after the three-round cap. Both normal agents returned NO_NEW_VALUABLE_FINDINGS in Round 3, but the risk-focused agent found a final FE-side scope correction that was independently verified and merged into the token round-trip comment; the review contract does not permit a fourth round. All currently known candidates are nevertheless adjudicated and included below.

Critical checkpoint conclusions:

  • Data correctness: failed. Session-token credentials are dropped/staled across FE DDL and meta-service paths, and Recycler exists status mapping can turn real provider failures into false not-found results.
  • Concurrency and lifecycle: delete-task ownership, executor waiting, batch clamping, and error propagation are sound; request admission during recursive deletion is not.
  • Configuration and dynamic behavior: Recycler rate limiting and PUT fault injection are bypassed for the actual recursive-delete SDK requests; AWS provider precedence, refresh-capable providers, and client cache identity otherwise remain compatible.
  • Compatibility and rolling behavior: the optional protobuf field is wire-compatible, but the Recycler 0/1/negative adapter contract and GCS iterator migration are broken.
  • Parallel paths: BE/Recycler and S3/Azure/GCS paths were traced; the GCS path has an unconditional compile failure and Recycler differs from the preserved BE admission behavior.
  • Tests and validation: no builds or tests were run, as required by the review prompt. Existing S3 accessor tests still require 1 for not-found, and there is no end-to-end token persistence/redaction/rotation coverage; the GCS compile error is statically evident.
  • Observability and security: the session token lacks SK-equivalent encryption/log/display handling, and successful S3 writes now log at INFO on the hot path. This is credential-secret handling within authenticated control paths; no unsupported cross-tenant vulnerability claim is made.
  • Persistence and recovery: token-bearing vault/stage records can either lose the token or retain it plaintext, so persistence round trips are not safe.
  • Performance: recursive deletion can evade Recycler request controls, while per-write INFO logging adds log I/O proportional to storage QPS; page and provider batch limits themselves are sound.

User focus: review_focus.txt supplied no additional focus, so the entire PR was reviewed without narrowing scope.

Comment thread cloud/src/recycler/s3_accessor.cpp
Comment thread gensrc/proto/cloud.proto Outdated
Comment thread cloud/src/recycler/s3_accessor.cpp
Comment thread common/cpp/client/obj_storage_client.cpp Outdated
Comment thread common/cpp/client/s3_obj_storage_backend.cpp Outdated
Comment thread cloud/src/recycler/s3_accessor.cpp
@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 0.00% (0/1) 🎉
Increment coverage report
Complete coverage report

@sollhui
sollhui force-pushed the agent/unify-be-recycler-obj-client branch from 92a3812 to 3f0e01e Compare August 4, 2026 03:18
@sollhui

sollhui commented Aug 4, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

2 similar comments
@sollhui

sollhui commented Aug 4, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

@sollhui

sollhui commented Aug 4, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

@sollhui

sollhui commented Aug 4, 2026

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review opinion

Request changes. The shared object-storage refactor is largely coherent, but it narrows the Azure exception boundary for multipart upload and HEAD. A standard exception from successful-response parsing can now escape the facade; on the BE upload pool that can terminate the process instead of completing the upload buffer with an error. I left one P1 inline comment.

Critical checkpoint conclusions

  1. Goal and proof: The PR consistently moves BE and Recycler S3/Azure behavior behind shared backends, authentication factories, admission control, and recursive-delete orchestration. The migration is mostly achieved, but the accepted Azure exception-boundary regression means the goal is not safely complete. The changed unit/mocked tests cover many facade, limiter, writer, credential, and recursive-delete paths, but no negative test proves conversion of non-RequestFailedException SDK failures.
  2. Focus and clarity: Although large, the change is focused on removing duplicate object clients and relocating their callers/tests. The new facade/backend split is understandable, and no unrelated production change survived the full review.
  3. Concurrency: I traced the BE upload pool, client cache/reset locking, request admission, and Recycler executor submission, bounded backpressure, cancellation, waits, and first-error settlement. Locks remain scoped around lightweight shared state and no new lock-order/deadlock issue was found. The blocker is the uncaught Azure standard exception crossing the upload-worker boundary.
  4. Lifecycle and static state: Backend/client/credential shared ownership, AWS SDK/static lifetime, executor reset, Azure shared-key/SAS lifetime, and cross-library bvar/CMake ownership were checked. No circular ownership, shutdown leak, or cross-TU initialization-order defect was found.
  5. Configuration: No new configuration item is introduced. Existing endpoint, TLS/CA, credential-provider, limiter, and internal-bucket settings are propagated through the new factories; no additional dynamic-reload defect was found.
  6. Compatibility: No storage format, persisted metadata, FE-BE protocol, or externally serialized value is changed. Numeric status conventions, namespace aliases, request fields, and provider behavior were traced; no rolling-upgrade compatibility issue was found.
  7. Parallel paths: S3/Azure/GCS, AWS V1/V2 and credential modes, BE/Recycler, synchronous/executor deletion, and reader/writer/list/delete/HEAD paths were compared. The accepted Azure upload_part/head_object boundary is the only uncovered distinct regression; other concerns are already in live threads or were dismissed with code evidence.
  8. Conditionals: Provider selection, BUILD_AZURE/USE_AZURE, NOT_FOUND mapping, page termination, capability batching, and credential branches were checked against siblings. No additional incorrect or unexplained special condition was found.
  9. Test coverage: The PR updates broad C++ unit/mock coverage, including the facade, rate limiting, writers, credentials, and recursive deletion. It lacks a negative test for an ordinary exception from Azure successful-response parsing; existing live review threads already cover other missing executor/provider cases. No new end-to-end regression test is added.
  10. Test results: No result/golden files are changed. I did not run local builds or tests under this review-only bundle. At submission time Cloud UT and FE UT pass; Clang Formatter fails; BE UT, macOS BE UT, compile, performance, and this review check are still pending.
  11. Observability: Provider errors, request IDs, HTTP status, TLS context, and bvars were traced before Recycler response collapse. Apart from concerns already raised in live threads, no additional production observability gap survived; the suspected Azure ListBlobs gap was dismissed because the pinned storage exception already includes provider message and request ID in what().
  12. Transactions and persistence: No EditLog, transaction-state, master-failover, or persistent-storage schema change is introduced.
  13. Data writes, atomicity, and crashes: Multipart PUT, completion, reads, batch deletes, recursive deletion, retry/idempotency, and partial-failure settlement were traced. No new atomicity issue was found, but the accepted Azure exception path can escape the upload worker and terminate the BE instead of returning an error, so this checkpoint is blocking.
  14. FE-BE variables: No new variable or thrift/protobuf value needs propagation between FE and BE.
  15. Performance: Page and provider batch sizes, bounded streaming deletion, admissions, cache reuse, allocations, and hot-path logging were reviewed. No additional substantiated CPU, memory, complexity, or redundant-work regression was found beyond concerns already covered in live threads.
  16. Other issues: Authentication and cloud trust-boundary changes were checked against SECURITY.md and the threat model; no new distinct security issue survived deduplication. No additional nullability, ownership, memory-safety, or stale-symbol defect was found.

Review completion and focus

The focus file contains no extra user guidance, so I reviewed the entire 64-file PR. Three full convergence rounds completed; all final reviewers returned NO_NEW_VALUABLE_FINDINGS against the same frozen one-comment set. Every candidate is accepted, merged, duplicated, already covered, or dismissed with concrete evidence. Immediately before submission, the live head/base still matched 5f7c022169e34d78936527cd3a653d2bb564f93b / 07ba0d1e448be58bef46e17c01498cb3d9864891, and no live review, inline, or issue comment duplicated the accepted finding.

Comment thread common/cpp/client/azure_obj_storage_backend.cpp
### What problem does this PR solve?

Issue Number: None

Related PR: apache#66350

Problem Summary: Azure multipart upload and HEAD response parsing could throw standard-library exceptions that escaped the object-storage facade and could terminate an upload worker. The Azure client factory aggregate also failed BE UT compilation under -Wmissing-designated-field-initializers. Catch and convert standard exceptions with path context, default-initialize all factory result fields, and merge the latest master while preserving the common object-client include in the embed tests.

### Release note

None

### Check List (For Author)

- Test: Not run at the author request; formatted modified C++ files with clang-format 16 and checked their diffs with git diff --check
- Behavior changed: Yes, Azure SDK response parsing failures now return INTERNAL_ERROR instead of escaping
- Does this need documentation: No
@sollhui

sollhui commented Aug 5, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

@sollhui

sollhui commented Aug 5, 2026

Copy link
Copy Markdown
Contributor Author

/review

@github-actions

github-actions Bot commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: All Codex review accounts are usage-limited; earliest retry is 2026-08-08T03:32:00Z.
Workflow run: https://github.com/apache/doris/actions/runs/30974256868

Please trigger /review again after that time.

### What problem does this PR solve?

Issue Number: None

Related PR: apache#66350

Problem Summary: The clang-format GitHub Actions job rejected the FileInfo aggregate initialization in S3FileSystem::list_impl. Apply the clang-format 16 layout expected by the repository formatter.

### Release note

None

### Check List (For Author)

- Test: Ran ./build-support/check-format.sh successfully; no compilation or runtime tests were needed for a formatting-only change
- Behavior changed: No
- Does this need documentation: No
@sollhui

sollhui commented Aug 5, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

@sollhui

sollhui commented Aug 5, 2026

Copy link
Copy Markdown
Contributor Author

/review

@hello-stephen

Copy link
Copy Markdown
Contributor

Cloud UT Coverage Report

Increment line coverage 49.13% (85/173) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 76.75% (1921/2503)
Line Coverage 64.11% (34423/53690)
Region Coverage 64.29% (17335/26963)
Branch Coverage 53.83% (9295/17266)

@github-actions

github-actions Bot commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: All Codex review accounts are usage-limited; earliest retry is 2026-08-08T03:32:00Z.
Workflow run: https://github.com/apache/doris/actions/runs/30975600673

Please trigger /review again after that time.

@sollhui

sollhui commented Aug 5, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

@sollhui

sollhui commented Aug 5, 2026

Copy link
Copy Markdown
Contributor Author

/review

@sollhui

sollhui commented Aug 5, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

@github-actions

github-actions Bot commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: All Codex review accounts are usage-limited; earliest retry is 2026-08-08T03:32:00Z.
Workflow run: https://github.com/apache/doris/actions/runs/30994127881

Please trigger /review again after that time.

@hello-stephen

Copy link
Copy Markdown
Contributor

Cloud UT Coverage Report

Increment line coverage 49.13% (85/173) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 76.75% (1921/2503)
Line Coverage 64.07% (34399/53690)
Region Coverage 64.25% (17323/26963)
Branch Coverage 53.80% (9289/17266)

### What problem does this PR solve?

Issue Number: None

Related PR: apache#66350

Problem Summary: BE UT compilation treats missing designated aggregate fields as errors. AwsCredentialFactory returns AwsCredentialResult with only provider or error initialized, causing seven -Wmissing-designated-field-initializers failures. Default-initialize both result fields so partial designated initializers are valid and consistent with the Azure credential result.

### Release note

None

### Check List (For Author)

- Test: Not run at the author request; formatted the modified header with clang-format 16 and checked with git diff --check
- Behavior changed: No
- Does this need documentation: No
@sollhui

sollhui commented Aug 5, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

### What problem does this PR solve?

Issue Number: None

Related PR: apache#66350

Problem Summary: BE UT builds with PCH disabled, and s3_client_factory_test.cpp directly uses io::ObjClientHolder without including the header that declares it. Add the direct s3_file_system.h include so the test translation unit compiles independently.

### Release note

None

### Check List (For Author)

- Test: Local BE UT was not run; clang-format 16 dry-run and git diff --check passed.

- Behavior changed: No

- Does this need documentation: No
@hello-stephen

Copy link
Copy Markdown
Contributor

Cloud UT Coverage Report

Increment line coverage 49.13% (85/173) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 76.75% (1921/2503)
Line Coverage 64.05% (34386/53690)
Region Coverage 64.23% (17317/26963)
Branch Coverage 53.77% (9284/17266)

@sollhui

sollhui commented Aug 5, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

1 similar comment
@sollhui

sollhui commented Aug 6, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 57.04% (77/135) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 59.70% (26042/43624)
Line Coverage 44.02% (263894/599482)
Region Coverage 39.78% (210249/528512)
Branch Coverage 41.20% (96179/233422)

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 72.06% (98/136) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 74.43% (31664/42540)
Line Coverage 58.76% (349902/595427)
Region Coverage 54.88% (291013/530234)
Branch Coverage 55.93% (130568/233446)

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 72.06% (98/136) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 74.43% (31662/42540)
Line Coverage 58.76% (349868/595427)
Region Coverage 54.88% (290990/530234)
Branch Coverage 55.92% (130552/233446)

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 28951 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d49c0a95c5cf5e5c6c2c95497ae0eaad3744b7cc, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17596	3980	3970	3970
q2	2020	340	210	210
q3	10415	1446	828	828
q4	4726	476	341	341
q5	7908	913	561	561
q6	269	169	136	136
q7	781	816	615	615
q8	10593	1654	1557	1557
q9	5718	4099	4119	4099
q10	6832	1639	1355	1355
q11	507	346	338	338
q12	752	584	460	460
q13	18142	3317	2733	2733
q14	261	259	235	235
q15	q16	736	734	665	665
q17	1041	947	1034	947
q18	6602	5632	5590	5590
q19	1177	1249	1113	1113
q20	797	647	595	595
q21	5550	2619	2312	2312
q22	420	350	291	291
Total cold run time: 102843 ms
Total hot run time: 28951 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4281	4178	4189	4178
q2	273	327	207	207
q3	4549	5017	4337	4337
q4	2206	2273	1395	1395
q5	4229	4111	4112	4111
q6	231	180	125	125
q7	1731	1564	1823	1564
q8	2617	2140	2075	2075
q9	7310	7262	7309	7262
q10	4329	4309	3913	3913
q11	546	398	371	371
q12	727	739	505	505
q13	3195	3667	2917	2917
q14	295	292	276	276
q15	q16	712	708	643	643
q17	1368	1324	1314	1314
q18	12195	11023	11881	11023
q19	1166	1175	1148	1148
q20	2226	2244	1943	1943
q21	5674	4952	4716	4716
q22	526	458	395	395
Total cold run time: 60386 ms
Total hot run time: 54418 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 166727 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d49c0a95c5cf5e5c6c2c95497ae0eaad3744b7cc, data reload: false

query5	4308	596	449	449
query6	458	218	201	201
query7	4854	550	340	340
query8	320	158	153	153
query9	8763	4079	4131	4079
query10	464	359	305	305
query11	5901	2168	2015	2015
query12	164	99	94	94
query13	1270	593	453	453
query14	6081	4245	3965	3965
query14_1	3769	3789	3736	3736
query15	198	192	177	177
query16	984	533	471	471
query17	906	676	534	534
query18	2420	479	334	334
query19	206	187	150	150
query20	105	99	99	99
query21	234	152	135	135
query22	13076	13062	12759	12759
query23	15802	14908	14635	14635
query23_1	14787	14852	14779	14779
query24	7607	1719	1243	1243
query24_1	1287	1263	1240	1240
query25	548	442	392	392
query26	1178	363	214	214
query27	2568	597	404	404
query28	4529	2054	2039	2039
query29	1091	638	489	489
query30	353	262	226	226
query31	1171	1121	1056	1056
query32	108	64	63	63
query33	535	316	245	245
query34	1169	1110	628	628
query35	737	751	637	637
query36	788	803	717	717
query37	163	112	96	96
query38	1844	1773	1695	1695
query39	813	833	791	791
query39_1	778	790	784	784
query40	255	188	151	151
query41	70	74	70	70
query42	97	98	97	97
query43	351	331	276	276
query44	1429	775	782	775
query45	182	174	172	172
query46	1106	1162	687	687
query47	1524	1549	1462	1462
query48	411	441	306	306
query49	579	401	292	292
query50	1107	423	353	353
query51	10391	10377	10349	10349
query52	85	85	75	75
query53	270	278	198	198
query54	294	229	238	229
query55	78	76	67	67
query56	298	315	289	289
query57	998	989	944	944
query58	298	261	256	256
query59	1555	1618	1409	1409
query60	306	275	247	247
query61	157	144	151	144
query62	387	317	273	273
query63	226	198	200	198
query64	2691	1025	886	886
query65	3933	3770	3837	3770
query66	1786	455	371	371
query67	28317	28144	27992	27992
query68	3225	1585	1087	1087
query69	400	303	252	252
query70	868	763	761	761
query71	376	330	320	320
query72	3046	2676	2383	2383
query73	827	762	441	441
query74	4624	4490	4306	4306
query75	2381	2337	1996	1996
query76	2305	1126	777	777
query77	339	366	267	267
query78	11187	11191	10540	10540
query79	1417	1095	720	720
query80	1290	578	499	499
query81	539	338	285	285
query82	619	177	147	147
query83	421	344	315	315
query84	330	167	138	138
query85	1090	701	585	585
query86	399	233	229	229
query87	1978	1963	1840	1840
query88	3729	2827	2853	2827
query89	395	320	278	278
query90	1923	203	196	196
query91	200	192	169	169
query92	65	64	53	53
query93	1565	1598	927	927
query94	727	360	319	319
query95	780	518	560	518
query96	1089	798	344	344
query97	2451	2474	2343	2343
query98	199	190	182	182
query99	724	725	612	612
Total cold run time: 253214 ms
Total hot run time: 166727 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 23.91 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d49c0a95c5cf5e5c6c2c95497ae0eaad3744b7cc, data reload: false

query1	0.00	0.00	0.00
query2	0.08	0.05	0.05
query3	0.25	0.14	0.13
query4	1.61	0.14	0.14
query5	0.23	0.22	0.23
query6	1.16	0.79	0.82
query7	0.04	0.01	0.01
query8	0.06	0.04	0.04
query9	0.36	0.31	0.30
query10	0.57	0.54	0.54
query11	0.18	0.14	0.13
query12	0.18	0.14	0.14
query13	0.47	0.47	0.46
query14	1.02	1.01	1.00
query15	0.61	0.57	0.59
query16	0.33	0.33	0.31
query17	1.10	1.09	1.06
query18	0.22	0.20	0.20
query19	2.11	1.96	2.01
query20	0.01	0.01	0.01
query21	15.45	0.21	0.13
query22	4.85	0.05	0.05
query23	16.13	0.30	0.13
query24	2.94	0.44	0.31
query25	0.11	0.05	0.04
query26	0.74	0.21	0.14
query27	0.04	0.04	0.04
query28	3.49	0.73	0.34
query29	12.48	4.02	3.22
query30	0.28	0.15	0.16
query31	2.77	0.57	0.31
query32	3.22	0.58	0.49
query33	3.15	3.26	3.25
query34	15.53	3.95	3.26
query35	3.23	3.20	3.21
query36	0.56	0.44	0.42
query37	0.09	0.07	0.07
query38	0.05	0.04	0.03
query39	0.04	0.03	0.03
query40	0.17	0.16	0.15
query41	0.08	0.03	0.03
query42	0.04	0.03	0.03
query43	0.04	0.04	0.03
Total cold run time: 96.07 s
Total hot run time: 23.91 s

@sollhui sollhui removed the dev/4.1.x label Aug 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants